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Outline 


Just  how  good  is  Agile? 

•  The  good,  the  bad,  and  the  ok. . .. 

Is  it  right  for  all  circumstances?  If  not,  when? 

•  Not  a  silver  bullet 

Must  it  be  done  in  a  “pure”  form?  If  not,  what  is  gained  and  what  is  lost? 

•  Maybe  so,  maybe  not 
What  must  I  do  to  be  successful 

•  Some  take  aways 
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JUST  HOW  GOOD  IS  AGILE 
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Empirical  Studies  on  Agile 


Studies  by  research  method* 

Company  studies  and 

Research  method 

Number 

Percent 

case  studies 

-  Single-case 

13 

39 

•  Microsoft  Research 

-  Multiple-case 

11 

33 

-  Survey 

4 

12 

•  Virginia  Polytechnical 

-  Experiment 

3 

9 

Institute  PhD  (2013) 

-  Mixed 

2 

6 

Workshops  and  Cross 

Total 

33 

100 

company  interviews 

Industry-performed  quantitative  studies  ’  SB  Agile  Collaborators 

Working  Group 

•  VersionOne  surveys 

•  NDIA/AFEI  ADAPT 

•  Rally  Software  Quantitative  Analysis 

•  CAST  CRASH  Report 

*T.  Dyba°  ,  T.  Dings0yr.  Empirical  studies  of  agile  software  development:  A  systematic  review, 

Information  and  Software  Technology  50  (2008)  833-859 
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It’s  a  Journey 


Patriot  Excalibur  started  in  2003  and  continues  today.... 
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Agile  Delivers  Even  with  Smaller  Team 
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Agile  Success  Comes  with  Challenges 
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CAST  Report  on  Application  Software  Health  (research.castsoftware.com) 
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CRASH  Report  -  201 1/12  •  Summary  of  Key 
Findings 

Figure  16b.  Transferability  Scores  Figure  16c.  Changeability  Scores 


CRASH  Report  -  201 1/12  •  Summary  of  Key 

Findings  Figure  16a.  Total  Quality  Index 

Scores  by  Development  Methods 

Finding  6 — Development 
Methods  Affect  Structural  Quality 


•  agile  methods  are  nearly  as 
effective  as  waterfall  at 
managing  the  structural 
quality  affecting  business  risk 
(Robustness,  Performance, 
and  Security) 

•  less  so  at  managing  the 
structural  quality  factors 
affecting  cost  (Transferability 
and  Changeability) 
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Custom 


Pitfalls  of  (Agile)  Measurement 


The  Seven  Deadly  Sins  of  Agile  Measurement 

■ 

Deadly  Sim 

Heavenly  Virtue 

1 

Using  metrics  as  levers 

Us i n g  m etri cs  f o r  f e e d b a ck  to  i m p rove  your 

to  change  someone 

own  performance 

else's  behavior 

2 

Unbalanced  metrics 

Day-one  have  one  metric  from  each  quadrant 

3 

Believing  metrics  can 

Use  quantitative  insightto  complementrather 

replace  thinking 

than  replace  qualitative  insight 

4 

Too  costly  metrics 

Favor  automatic  metrics  from  passively 

acquired  data  or  lightweight  surveys 

5 

Using  a  convenient 

U  s  e  0  D 1 M  to  d  ete  rm  i  n  e  m  etri  cs  th  e  p  rovi  d  e 

metric 

critical  insight  and  drive  to  your  desired 

outcomes 

6 

Using  bad  analysis 

Get  your  statistics  right  by  consulting  experts 

7 

Forecasting  without 

Use  the  percentile  coverage  distribution,  the 

discussing  probability 

cone  of  uncertainty,  or  Monte  Carlo  simulation 

#F?3 1  lyON13  giLMacch  ercn e 

LMkcch =ron e@ral  lydev.com  can  u  Raiiy  sabt/wt  c wicker:  cap 

=  Software  Engineering  Institute  CarnegieMellon 


)2013  Carnegie  Mellon  University 


Some  Common  Agile  Myths 


Mythbusters 


Myth 

I 

S3. 

1 

■Quality 

Predictability 

Productivity 

Points  +  hours  better  than  points  alone 

Dedicate  to  one  team 

Keep-  the  teams  stable 

Lower  Wl  F  is  always  better 

Ideal  team  size:  5-9 

SiultfptrfNHti  wwn  bwS  ikMt  futiw 

Kanban  is  better  than  Sou  m 

SoumBan  is  the  best  of  both  woHds 

Busted 


Confirmed 


Minimal  difference 


WfP  #Ral lyON  1 3  @Ulacch econ  e  LMaoch Ercs ncgira I  lydEV.com  sac  13.  fsbIi>  £c«wart  Deweopnen:  cap 
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Federal  Challenges  in  Applying  Agile 


GAO  2012  report  of  experiences  in  5  agencies 

•  32  Agile  practices  identified  for  consideration 

•  10  practices  were  used  and  deemed  effective 

•  14  challenges  were  identified  reflecting  on  the  need  to  transition 

•  Team  transition  issues 

•  Guidance  and  adoption  of  tools  were  difficult 

•  Agency  commitment  of  staff 

•  Customer  trust  of  iterative  solutions 

•  Adapting  to  iteration  time  frames  was  difficult 

•  Federal  reporting  and  reviews  not  aligned  with  Agile 
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IS  AGILE  RIGHT  FOR  ALL? 
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Dynamic  Environments  -  Traditional  versus 
Agile  Worlds 


. the  Traditional  World  struggles  to  deliver  as  it  constantly  looks  back 

at  long-fixed  requirements  and  priorities. 

•  If  requirements  are  stable,  then  safer  and  more  prudent  to  use 
waterfall 


. the  Agile  World  adapts  as  it  delivers  by  constantly  looking  forward  at 

evolving  requirements  and  priorities. 

•  In  settings  with  significant  operational  or  technology  dynamism,  the 
Agile  methods  are  an  advantage 


Parallel  Worlds:  Agile  and  Waterfall  Differences  and  Similarities  (CMU/SEI-2013-TN-021). 
http://resources.sei.cmu.edu/library/asset-view.cfm?AssetlD=62901 
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Microsoft  Research  Agile  Trends  (2013) 


Did  non-afiile  practiti □  nee rs  previously  work  on 

ag.No  projects,? 
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Comparison  of  Agile  Benefits 

Microsoft  Research 

Agile  Benefits 


A^iTe  Oevs'  Perceived  Benefits 

Norr  Agile  Devs:-  Perceived  Benefits 
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Some  Alleged  Agile  Problems 


M  Eero  5  oft  R  esearch 


Agile  Perceived  Problems 
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Have  Agile  Techniques  been  the  Silver  Bullet 
for  Software  Development  at  Microsoft? 

Results 

•  the  growth  of  agile  adoption  at  Microsoft  is  slower  than  expected 

•  no  individual  agile  practice  exhibited  strong  growth  trends 

•  both  agile  and  non-agile  practitioners  agreed  on  the  relative  benefits  and 
problem  areas  of  agile  techniques 


Conclusions 

•  no  clear  trends  in  practice  adoption 

•  non-agile  practitioners  are  less  enamored  of  the  benefits  and  more 
strongly  in  agreement  with  the  problem  areas 

•  the  ability  for  agile  practices  to  be  used  by  large-scale  teams  generally 
concerned  all  respondents 
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Traditional  vs  Agile  Approaches  Fit, 


Traditional  approach 

•  consistent  with  the  acquisition  life  cycle  guidance  provided  in  the  DoD 
Acquisition  Deskbook  and  its  supporting  documents. 

•  programs  with  stable  requirements  and  environment,  with  known 
solutions  to  the  requirements 

•  programs  with  a  homogeneous  set  of  stakeholders  who  communicate 
well  via  documents 

•  programs  for  which  the  technology  base  is  evolving  slowly 
(technology  is  not  expected  to  be  refreshed/replaced  within  the 
timeframe  of  the  initial  development) 
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Traditional  vs  Agile  Approaches  Fit2 


Agile  approach 

•  programs  with  volatile  requirements  and  environment 

•  programs  where  solutions  are  sufficiently  unknown  that  significant 
experimentation  is  likely  to  be  needed 

•  programs  for  which  the  technology  base  is  evolving  rapidly 

•  programs  with  stakeholders  who  can  engage  with  developers  in 
ongoing,  close  collaboration 


concluded  that,  in  reality,  no  acquisition  context  that  we  have  seen  is  “ideal”  for 

either  the  traditional  or  agile  approach. 
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MUST  IT  BE  DONE  IN  “PURE 
FORM 
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Capabilities  delivered  Capabilities  delivered 


Both  Waterfall  and  Agile  Development  Have 
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Agile  Development 


Cost  of  over  analysis,  up-front  requirements, 
design  delays  capabilities  delivered,  creates 
missed  opportunities 


tegrated  Approach 
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Assess  the  impact  of: 

•  delivered  capabilities 

•  cost  of  delay,  rework 
to  determine  efficient 

increments. 

i 

Accumulated  suboptimal  architecture,  lack  of 
communication  and  clear  requirements  impact 
capabilities  delivered.  The  consequences  are 
delays,  defects  and  inability  to  deliver 
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What  about  modifying  SCRUM? 
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Microsoft  Research  Practices 

Agile  Devs  Team  Practices 
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User  stories 
Sustainable  pace 
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Pair  programming 
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Agile  Techniques:  Silver  Bullet? 


I 

1 

I  - 

1 

- r  i 

i  i 

i  i 

i 

i 

i  i 

r  i 

i  i 

i  i  i 

i  i  i 

. i .  i  i 

!  1  1 

0%  20%  40%  60%  80%  100% 


■  No 

■  Yes 


Software  Engineering  Institute  CarnegieMellon 


©2013  Carnegie  Mellon  University 


25 


Use  of  Agile  Techniques 


AGILE  TECHNIQUES  EMPLOYED 
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*  Daily  Standup 
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Continuous  Integration 
Automated  Builds 
K  Dedicated  Product  Owner 

•  Integrated  Dev/QA 
^  Refactoring 
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}>  Digital  Taskboard 
Story  Mapping 
>■  Kanban 
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Scaling  Agile  Brings  in  More  Variation 


Scaled  Agile  Framework 

•  Kanban,  SCRUM,  Value  Stream  Mapping 

Disciplined  Agile  Delivery 

•  RUP,  XP,  SCRUM 

DSDM 

•  Popular  scaling  approach  in  Europe 


https://www.thecsiac.com/spruce/resources/ref_documents/agile-scale-aas-spruce-sei 
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It’s  not  about  the  practices  and  methods 

It’s  about  the  principles 


Highest  priority  is  satisfy  the 
customer  through  early  and 
continuous  delivery  of  software. 

Welcome  changing  requirements, 
even  late  in  development. 

Deliver  working  software 
frequently,  from  a  couple  of  weeks 
to  a  couple  of  months. 

Business  people  and  developers 
must  work  together  daily 
throughout  the  project. 

Build  projects  around  motivated 
individuals.  Provide  environment 
and  support  they  need. 

The  most  efficient  and  effective 
method  of  conveying  information 
to  and  within  a  development  team 
is  face-to-face  conversation. 


7.  Working  software  is  the  primary 
measure  of  progress. 

8.  Agile  processes  promote  sustainable 
development... a  constant  pace 
indefinitely. 

9.  Continuous  attention  to  technical 
excellence  and  good  design 
enhances  agility. 

10.  Simplicity— the  art  of  maximizing  the 
amount  of  work  not  done--is 
essential. 

11.  The  best  architectures, 
requirements,  and  designs  emerge 
from  self-organizing  teams. 

12.  At  regular  intervals,  the  team  reflects 
on  how  to  become  more  effective, 
then  tunes  and  adjusts  its  behavior 
accordingly. 
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WHAT  MUST  I  DO  TO  BE 
SUCCESSFUL 
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Understand  your 
organization’s 
alignment  with  Agile 
principles  and 
practices 
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Traditional  approach 


Strengths  of  the  traditional  approach  include: 

•  enables  the  comparability  and  repeatability  that  standardization  provides 

•  enables  a  contractually  verifiable  definition  of  completed  intermediate  work 
products 

•  reduces  risks  by  means  of  contractually  assured  baselines 

Weaknesses  of  the  traditional  approach  include: 

•  the  process  drives  measurement  of  compliance  with  itself  as  a  primary 
measure  of  success  (i.e.,  rather  than  measuring  success  as  deploying  a 
workable  solution) 

•  it  depends  on  documents  as  the  basis  to  verify  and  validate  the  requirements, 
the  architecture,  and  the  detailed  design 

•  most  of  the  requirements  are  completed  before  any  code  is  written,  thus 
extending  development  timelines 
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Agile  approach 

Strengths  of  this  approach  include 

•  early  insight  by  the  users  into  the  shape  of  the  solution 

•  early  course  correction 

•  “fail  fast”  (If  the  early  solution  ideas  turn  out  to  be  flawed,  little  time  or 
money  is  spent  before  that  learning  occurs.) 

•  explicit  understanding  that  the  requirements  are  expected  to  evolve 

Weaknesses  of  this  approach  (particularly  in  large  acquisition  settings) 
include 

•  more  dependence  on  tacit  knowledge  (e.g.,  lack  of  explicit 
documentation)  as  the  basis  for  decision-making  than  is  comfortable 
for  most  acquisition  organizations 

•  dependence  on  availability  of  actively  engaged  user/customers 

•  difficulty  in  aligning  implementation-driven  artifacts  and  measures  with 
those  of  the  larger  traditional  acquisition  setting. 
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Effective  Agile  Practices  in  Federal  Settings 


Table  1:  Practices  Used  and  Found  Effective  by  Five  Agencies 


Practice 


1.  Start  wi'.h  Agile  guicance  and  an  Agile  adoption  strategy. 


2.  Enhance  migration  to  Ag  e  concepts  using  Ag  le  terms  and  examples. 

3.  Continuously  improve  Agile  adoption  at  Iboth  project  and  organization  levels. 

4.  Seek  to  identify  and  address  impediments  at  the  organization  and  project  levels. 

5.  Obtain  stakehclder/customer  feedback  frequently  and  closely. 

0.  Empowe'  smalL  crass-functional  teams, 

7.  Induce  'equipments  related  to  securty  and  progress  monitoring  in  your  queue  of 
unfinished  work  (bade  Jog 

3.  Ga  n  trust  by  demonstrating  value  at  the  end  of  each  iteration. 

9.  Track  progress  using  tools  and  metrics. 

10.  Track  prog'ess  da  ly  and  visibly. 

■&r:  GAO 


GAO-12-681  Agile  Effective  Practices  and  Federal  Challenges 
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Successful  Management  Traits  within  Agile 
Teams 

Executing  Side  (developer)  Acquiring  Side  (PMO) 


Leader  -  more  time  with  team  than 
behind  the  office  desk 

Coach  -  seed  team  with  ideas  and 
allow  them  to  solve  the  problem 

Expeditor  -  help  remove  operational 
impediments 

Champion  -  communicate  with  upper- 
level  management  and  stakeholders 
(translator  role) 

Ambassador  -  cultivate  relationships 
with  end  users  and  subject  matter 
experts  and  their  management 


Leader  -  establish  and  maintain 
relationships  with  executing  group 

Coach  -  help  existing  personnel  make 
transition  to  fast-tempo,  high- 
interaction  environment  of  Agile 

Expeditor  -  efficiently  deploy  people 
interacting  with  development  team. 

Champion  -  maintain  buy-in  from 
external  funders  and  stakeholders 

Ambassador  -  ensure  appointment  of 
end  users  or  SMEs  to  work  with 
developers. 

Agile  Methods:  Selected  DoD  Management  and  Acquisition  Concerns 

http://www.sei.cmu.edU/librarv/abstracts/reports/1 1tn002.cfm?DCSext. 

abstractsource=SearchResults 
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Bottom  Line  Take  Aways 


Accumulated  empirical  evidence  is  scant  but  increasing.  Evidence 
shows: 

•  Agile  can  be  effective 

•  Agile  is  not  a  silver  bullet 

•  Agile  is  not  conducive  to  every  situation 

•  Agile  is  a  different  mindset  and  requires  trust 

•  Agile  requires  planning  and  hard  work 
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For  More  Information,  or  to  Join  SEI’s  Agile 
Collaboration  Group,  Contact... 

Mary  Ann  Lapham 

Principal  Engineer 
Telephone:  412-268-5498 
Email:  mlapham@sei.cmu.edu 

U.S.  Mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-3890 
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